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A voice and pointer operated system includes a pen (12) provided with a microphone (20) responsive to a voice command for 
pmvid^g a voice sfgnal andTe^ctronic device (14) having a pointer-responsive areata the form of a touch screen 06) ^ereon for 
LvJ^na with thP nen (12) as well as apparatus for recognizing the voice signal and producing an action routine denoted thereby. The 

frames '<* interaction with the pen (12) to select function for the 
SSSra entered via ^microphone (20) in the pen being used to provide input data to the electronic 

device (14). The pen (12) may be wireless or electrically coupled to the electronic device (14). 
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VOICE /POINTER OPERATED SYSTEM 
Background of the Invention 

1. Field of the Invention. 

The present invention relates to systems for 
5 entering data into a device using voice recognition of 
spoken words, and more particularly to systems capable 
of combining voice recognition with other forms of data 
entry such as by touching. 

2 . History of the Prior Art . 

10 it is well known to provide an electronic device 

with a pointer member such as a pen or stick for 
purposes of inputting data into the device. Typically, 
the pen or stick is used to activate buttons and to 
highlight fields displayed on touch- screen displays. 

15 In addition, the pen or stick may be used to perform 
other functions, such as handwriting recognition . In 
such arrangements, the pen or stick provides a means of 
human interfacing whereby the operator may point to 
fields, frames or buttons to make selections or 

20 otherwise enter data into an electronic device, such as 
a personal computer, a personal organizer or a personal 
digital assistant (PDA) . Additional data, such as that 
entered for purposes of recording or storage, is often 
entered by other interfacing means such as a keyboard 

25 or keypad. 

Still other arrangements for inputting data into an 
electronic device are known. Among such arrangements 
are those which use voice recognition techniques to 
recognize voice commands entered by the user. Such 

30 electronic devices include systems in which remote 
controls are provided with sophisticated voice 
recognition electronics- The remote controls recognize 
spoken commands, translate the commands into the 
traditional remote digital control signals, and 

35 transmit the control signals to a controlled device. 
Examples of such systems are provided by co- pending 
application Serial No. 07/915,112 of Bissonnette et 



mr.irv <wn QS25326A1 I > 



WO 95/25326 PCIYUS95/02921 



- 2 - 

al . , entitled Voice Operated Remote Control Device, by 
co-pending application Serial No. 07/915,938 of 
Bissonnette et al . , entitled Voice Recognition 
Apparatus and Method, and by co- pending application 
5 Serial No. 07/915,114 of Fischer, entitled Remote 

Control Device. All three applications were filed on 
July 17, 1992 and are commonly assigned with the 
present application. 

A further example of an electronic device in which 
10 data is inputted using voice recognition is provided by 
co-pending application Serial No. 08/113,394 of Fischer 
et al., entitled Voice Operated Remote Control System, 
which application was filed August 27, 1993 and is 
commonly assigned with the present application. The 
15 Fischer et al . application describes a voice operated 
remote control system in which a remote control device 
responsive to the voice commands of a user transmits 
representations of the voice commands to a controlled 
device which then produces voice signals in response to 
2 0 the transmitted representations. The controlled device 
includes voice recognition circuitry for recognizing 
the transmitted voice commands and executing action 
routines denoted thereby. The remote control device 
receives the voice commands via a microphone and 
25 produces a corresponding analog audio signal which is 
modulated and then transmitted by an infrared 
transmitter. An audio receiver at the controlled 
device includes an infrared sensor for receiving the 
transmitted signal, and additional circuitry for 
demodulating and processing the transmitted signal into 
a corresponding voice signal. The voice signal is used 
to generate an incoming digital voice template for 
comparison with a plurality of digital reference voice 
templates. If a substantial equivalent is found, a 
35 corresponding action routine is executed to achieve a 
desired action within the controlled device. 
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Another electronic device utilizing voice 
recognition for data inputting is described by co- 
pending application Serial No. 08/134,327 of 
Bissonnette et al . , entitled Voice Activated Personal 
Organizer, which application was filed October 12, 1993 
and is commonly assigned with the present application. 
The Bissonnette et al. application describes a small, 
portable, hand-held electronic personal organizer which 
performs voice recognition on words spoken by a user to 
input data into the organizer, and records voice 
messages from the user. The spoken words and the voice 
messages are input via a microphone, and the voice 
messages are compressed and then converted into digital 
signals for storage. The stored digital voice messages 
15 are reconverted into analog signals and then expanded 
for reproduction using a speaker. The organizer is 
capable of a number of different functions, including 
voice training, memo record, reminder, manual reminder, 
time setting, message review, waiting message, 
20 calendar, phone group select, number retrieval, add 
phone number, security, and "no" logic. During such 
functions, data is principally entered by voice and 
occasionally through use of a limited keypad, and voice 
recordings are made and played back as appropriate . A 
25 visual display provides feedback to the user. During 

the various functions, the user can edit different data 
within the organizer by eliminating or correcting such 
data or entering new data. 

While the electronic devices described in the 
30 above-noted co-pending applications incorporate a 

number of different advantageous features for human 
interfacing with such electronic devices so that data 
can be efficiently and effectively transferred, it may 
be desired for certain applications to provide further 
35 improvements in human interfacing with such devices. 

For example, it would be desirable to be able to select 
functions or otherwise input data via manual means 
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while simultaneously entering data specific to those 
functions by non-manual entries such as by vocal 
commands . Such an arrangement would provide an easy, 
intuitive human interface, allowing for both 
5 simultaneous and independent speech and touch 
inputting . 

Brief Summary of the Invention 
Briefly stated, the present invention provides a 
voice and pointer operated system in which a pen or 
stick responsive to voice commands is operative to 
input data into a device using voice recognition, in 
addition to data inputted into the device using a 
pointer or tip region of the pen or stick. The device 
has a pointer-responsive area thereon as well as voice 
15 recognition means for recognizing a voice signal from 
the pen and producing an action routine denoted 
thereby. The pointer- responsive area may comprise a 
touch screen presenting frames for interaction with the 
pen to select functions for the device, with voice 
commands entered via a microphone in the pen being used 
to provide input data to the device. 

In a first embodiment of a voice and pointer 
operated system in accordance with the invention, the 
pen is electrically coupled to the device to provide 
25 the voice signal directly to the device. A wire 

extending between the pen and the device, and which is 
long enough to permit movement of the tip region of the 
pen to different areas of the touch screen, couples the 
microphone directly to the device to provide the voice 
30 signals thereto. The wire connection also enables the 
pen to utilize the power supply of the device. 

In a second embodiment of a voice and pointer 
operated system, the pen is wireless, thereby 
eliminating the need for a wire or other direct 
35 connection between the pen and the device. Voice 
signals generated by the microphone of the pen in 
response to voice commands from the user are modulated 



20 



:iI><WO 9S25326A1 I > 



WO 95/25326 



YUS9S/02921 



before being transmitted by the pen to an audio 
receiver within the device for conversion into the 
voice signal. The modulated voice signal may be 
transmitted by an infrared transmitter, within the pen, 
5 to an infrared sensor at the audio receiver within tbie 
device . 

The voice recognition apparatus within the device 
may assume any appropriate form. In one example, such 
apparatus includes a reference memory for storing a 

10 plurality of reference voice templates, a program 

memory for storing a control program, and a processor 
for generating an incoming voice template in response 
to each voice signal which is received, in order to 
produce a voice signal corresponding to the transmitted 

15 representation. The control program is executed to 
determine whether the incoming voice template is 
substantially equivalent to one of the reference voice 
tenplates. One of a plurality of action routines is 
then selected, based on the reference voice template 

20 which the incoming voice template matches. 

The device, which is controlled by the voice and 
pointer inputs of the pen or stick, may comprise such 
electronic devices as a computer, a personal organizer 
or a personal digital assistant (PDA) . The device may, 

25 for example, comprise a personal digital assistant 

which is organized to operate as a phone directory for 
storing names, addresses, phone numbers and other 
information. In that -event, various fields within a 
display on the phone directory are touched by the pen 

30 to select basic functions such as entry of an 

additional name or entry of digital information for a 
name already displayed, with the user then speaking 
into. the pen to enter the specific data within such 
fields. 

35 Brief Description Of The Drawings 

The foregoing and other objects, features and 
advantages of the invention will be apparent from the 
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following more particular description of preferred 
embodiments of the invention, as illustrated in the 
accompanying drawings, in which: 

Fig, 1 is a perspective view of a pen and a 
5 personal digital assistant, providing an example of a 
voice and pointer operated system in accordance with 
the invention ; 

Fig. 2 is a block diagram of a first embodiment of 
the system of Fig. 1 in which the pen is directly 
10 coupled by wire to the personal digital assistant; 

Fig. 3 is an alternative embodiment of the system 
of Fig. l in which the pen is wireless; 

Fig. 4 is a block diagram of the electronic portion 
of the pen of Fig. 3; 
15 Fig. 5 is a block diagram of the audio receiver 

forming a part of the personal digital assistant of 
Fig. 3; 

Fig. 6 is a basic flow chart illustrating the 
operation of the system of Fig. 1; and 
20 Fig. 7 is a detailed flow chart illustrating a 

detailed example of the operation of the system of Fig. 
1. 

Detail ed Description 
Fig. 1 shows a voice and pointer operated system 10 

25 in accordance with the invention. The system 10 of 

Fig. 1 includes a pen 12 and electronic device 14 with 
which the pen 12 is used. In the example of Fig. l, 
the device 14 comprises a personal digital assistant 
(PDA) . It should be appreciated, however, that systems 

30 in accordance with the invention may utilize other 
electronic devices such as computers and personal 
organizers which are adapted for voice and pointer 
operation, using the pen 12, in accordance with the 
invention. 

35 The electronic device 14 includes a pointer- 

responsive area thereon in the form of a touch- screen 
16. The pen 12 includes a pointer or tip region 18 
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thereof. Placement of the pen 12 to engage the touch- 
screen 16 by the tip region 18 results in the inputting 
of data into the touch- screen 16 in conventional 
fashion. In this connection, it should be understood 
5 that arrangements using other than physical touching 

can be used in accordance with the invention, including 
those in which the pointer-responsive area responds to 
infrared signals blocked by the pen 12, Any 
appropriate arrangement can be used in which the tip 
10 region 18 of the pen 12 interacts with the touch-screen 
16 or other pointer- responsive area of the device 14 to 
input data therein, when placed in close proximity 
thereto . 

In accordance with the invention, the pen 12 is 
15 provided with a microphone 20 mounted at an end of the 
pen 12 opposite the tip region 18. As the user speaks 
voice commands into the microphone 20, corresponding 
voice signals are provided to the device 14. The user 
presses a push-to-talk switch 21 on the pen 12 to 
20 signal the device 14 that voice commands are being 
transmitted thereto. As described hereafter, the 
device 14 utilizes voice recognition to input further 
data represented by the voice signals. 

The pen 12 may be directly coupled to the device 14 
25 by wire, as described hereafter in connection with Fig. 
2 . Alternatively, the pen 12 may be wireless and may 
transmit the voice commands to the device 14, such as 
by use of infrared techniques, as described hereafter 
in connection with Fig. 3. The manner in which the 
30 voice signals received by the microphone 20 of the pen 
12 are conveyed to the device 14 is unimportant . What 
is important is that the system 10 of Fig. 1, as is 
true of systems in accordance with the invention, be 
capable of inputting data into the device 14 using both 
35 voice commands and pointing. Moreover, the voice 

commands and the pointing can be utilized to input data 
into the device 14 simultaneously. 
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It will be appreciated by those skilled in the art, 
particularly from the detailed description to follow, 
that systems in accordance with the invention such as 
the system 10 shown in Pig. 1 provide a greatly 
5 improved, and indeed a natural, human interface of the 
user with the device 14. The pen 12 can be used to 
point to a desired field or other area of the device 
14, and thereby input data into the device 14 while 
simultaneously responding to the user' s voice commands 

10 to input additional data into the device 14 in 

conjunction therewith. This enables the user to point 
to a desired field or frame of the touch- screen 16 
within the device 14 while at the same time making 
choices and otherwise inputting data in connection with 

15 such field or frame using voice commands. 

Fig. 2 shows an embodiment of the system 10 of Fig. 
1 in which the pen 12 is directly coupled by wire to 
the device 14. As shown in Fig. 2, the microphone 20 
of the pen 12 is coupled by a wire 22 to the device 14. 

20 The wire 22, which is long enough to provide the user 
with flexibility in pointing to different portions of 
the touch-screen 16, provides the voice signals which 
are generated by the microphone 20 in response to the 
voice commands of the user directly to a 

25 microcontroller 24 within the device 14 . The user 

presses the push-to-talk switch 21 before speaking to 
signal the device 14 that voice signals are being 
transmitted thereto. 

The microcontroller 24 forms a part of a voice 

30 recognition circuit 26, and includes an analog- to- 
digital converter (A/D) 28. The microphone 20 of the 
pen 12 converts words spoken by the user into analog 
voice signals, and the A/D converter 28 converts such 
signals into corresponding digital signals . The 

35 microcontroller 24 also includes a microprocessor (MP) 
30 coupled to the A/D converter 28, and having access 
to both a read only memory (ROM) 32 and a random access 



WO 95/25326 



7US95/02921 



- 9 - 

memory (RAM) 34. The microcontroller 24 is coupled to 
an external DRAM 36, which forms a part of the voice 
recognition circuit 26, and which is utilized for voice 
recognition storage requirements, including primarily 

5 the storage of voice templates. 

The voice recognition apparatus shown and described 
in Fig. 2 is by way of example, and it should be 
understood that other apparatus capable of voice 
recognition can be used in accordance with the 

10 invention. For example, a digital signal processor 
(DSP) can be used to perform the voice recognition. 

The microcontroller 24 comprising the voice 
recognition circuit 26 operates in the same manner as 
described in the previously referred to co-pending 

15 applications, Serial Nos. 07/915,112, 07/915 ,938 and 

07/915,114- Such applications are incorporated herein 
by reference. As described in detail in co-pending 
application Serial No. 07/915,112, for example, the A/D 
converter 28 of Fig. 2 may comprise an 8-bit converter 

20 which samples incoming data at a preassigned frequency 
such as 9.6 KHz. The A/D converter 28 outputs a 
digital signal representing the input analog voice 
signal from the microphone 20. The microprocessor 30 
processes the digital voice signal together with a 

25 voice recognition software routine forming part of a 
control program stored in the ROM 32. The digital 
voice signal is converted into an incoming voice 
template that is compared against previously stored 
voice templates of the user's voice stored in the 

30 external DRAM 36. The program decodes the voice 

templates. Together with the external DRAM 36, the RAM 
34 comprises a reference memory for temporary storage 
of data. 

Thus, the analog voice signal from the microphone 
35 20 of the pen 12 is applied to the A/D converter 28 for 
conversion into an incoming digital voice signal . The 
reference memory, comprised of the external DRAM 36 in 
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conjunction with the RAM 34, stores a plurality of 
reference digital voice templates. The ROM 32 stores 
the control program. The microprocessor 30, which is 
coupled to the A/D converter 28, the ROM 32 and the RAM 
5 34, generates an incoming digital voice template from 
the incoming digital voice signal at the output of the 
A/D converter 28. The microprocessor 30 then executes 
the control program to determine whether the incoming 
digital voice template is substantially equivalent to 

10 one of the reference digital voice templates stored in 
the reference memory comprised of the RAM 34 and the 
external DRAM 36. The microprocessor 30 determines 
what action to take corresponding to a reference 
digital voice template, if the incoming digital voice 

15 template is found to have substantial similarity to the 
reference digital voice template. Again, however, it 
should be understood that this description is by way of 
example only, and that other voice recognition 
techniques can be used in accordance with the 

2 0 invention. 

The electronic device 14 includes a keyboard input 
38. The keyboard input 38 is not essential but may be 
used, as in the present example, for manual entry of 
additional data into the microcontroller 24 . The 
25 electronic device 14 also includes an audio circuit 40 
coupled through a speaker or speakers 42 to provide 
audible messages to the user of the device 14. 

The electronic device 14 of Fig. 2 includes a 
liquid crystal display (LCD) 44 coupled to the 

3 0 microcontroller 24 through an LCD display circuit 46. 

The LCD display 44, which provides a visual display to 
the user, forms part of the touch-screen 16 of the 
electronic device 14. The touch-screen 16 is 
responsive to the tip region 18 of the pen 12 to input 
3 5 data, with such data being provided to the 

microcontroller 24 by the LCD display circuit 46. The 
physical or touching relationship between the tip 
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region 18 of the pen 12 and the touch- screen of the LCD 
display 44 is represented by a dotted line 48 in Fig. 
2 . 

The touch- screen 16 of the device 14 responds to 
5 the tip region 18 of the pen 12 to input data into the 
device 14 in conventional pen or stick fashion. In 
addition, however, and in accordance with the 
invention, data is also input into the device 14 using 
the microphone 20 of the pen 12 and the voice 

10 recognition capabilities of the device 14. As 

previously described, the microphone 2 0 responds to 
vocal commands of the user by providing the analog 
voice signals to the microcontroller 24 comprising a 
portion of the voice recognition circuit 26. The 

15 microcontroller 24 attempts to recognize the voice 

signals, and when such recognition occurs additional 
data is input into the system. In this fashion, data 
inputted into the device 14 by voice command is 
combined with data input by touching the touch- screen 

20 16. 

Fig. 3 shows an alternative embodiment in which the 
pen 12 is not coupled by the wire 22 to the device 14 , 
as in the case of Fig. 2. Instead, the pen 12 is 
wireless and transmits representations of the voice 

25 commands of the user to an audio receiver 50 within ttie 
device 14. The audio receiver 50 in turn converts the 
transmitted signals into the voice signals for 
application to the microcontroller 24. The 
microcontroller 24 performs voice recognition on the 

30 voice signals in the same manner as described in 
connection with Fig. 2. 

Thus, the device 14 of Fig. 3 is like that of Fig. 
2, except for the presence of the audio receiver 50. 
The pen 12 of Fig. 3 is like the pen of Fig. 2, except 

35 that it transmits the vocal commands to the audio 

receiver 50 in wireless fashion. Because the pen 12 of 
Fig. 3 is not directly coupled to the device 14 of Fig. 



OC\Z>. <WO 9S25326A1 J_> 



10 



WO 95/25326 PCT/US95/02921 



- 12 - 

3 by wire, the pen 12 must have its own power supply. 
To conserve on power, the push- to- talk switch 21 which 
is mounted on the side of the pen 12 is used to apply 
power to the microphone 20 whenever the user wishes to 
speak into the microphone 20. As in the case of Pig. 
2, the tip region 18 of the pen 12 of Fig. 3 physically 
interacts with the touch- screen 16 formed in 
conjunction with the LCD display 44, as represented by 
the dotted line 48. 

The electronic portion of the pen 12 of Fig. 3 is 
shown in Fig. 4. As previously described, the 
microphone 20 responds to words spoken by the user to 
produce the analog voice signals . Such analog voice 
signals are amplified in an audio amplifier 54. The 
15 audio amplifier 54 conditions the signal for proper 

application to a voltage controlled oscillator (VCO) 56 
to frequency modulate (FM) a signal at the output of 
the VCO 56 in accordance with the analog audio signal 
from the amplifier 54. The output of the VCO 56 is 
20 conditioned by a driver circuit 58 prior to driving an 
infrared (IR) transmitter 60. 

The pen 12 is powered by batteries 62. Because 
there is no memory or microprocessor in the pen 12, 
there is no need to keep the pen 12 powered when not in 
25 use. Accordingly, the push- to- talk switch 21 is used 
to provide the power from the batteries 62 only when 
the user wishes to speak into the microphone 20. 
Pressing the push- to -talk switch 21 at the outer 
surface of the pen 12 couples the batteries 62 to a 
30 voltage regulator 64. The voltage regulator 64, which 
is coupled between a common ground line 66 and the 
push- to- talk switch 21, has an output terminal 68 
thereof coupled to provide a power supply voltage (+V) 
at a terminal 70. This regulated supply voltage is 
35 applied to the microphone 20, the audio amplifier 54 

and the VCO 56. The batteries 62 are directly coupled 
to the driver 58 by the push-to- talk switch 21. The 
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regulated supply voltage +V optimizes the operating 
conditions of the VCO 56. 

Whenever the user wishes to transmit a voice 
command to the electronic device 14, the user pushes 
5 the push- to- talk switch 21 and speaks the voice command 
into the microphone 20 of the pen 12. The resulting 
analog audio signal is amplified by the amplifier 54 
and applied to the VCO 56 to frequency modulate the 
output of the VCO 56. This FM signal is applied via 
10 the driver 58 to the infrared transmitter 60, for 

transmission as an infrared signal to the electronic 
device 14. When the user is not transmitting voice 
commands, the push-to-talk switch 21 is open, and the 
batteries 62 are not coupled to the voltage regulator 
15 64 or to the driver 58. 

When the user pushes the push-to-talk switch 21 and 
speaks a command word into the microphone 20, the pen 
12 transmits a frequency modulated infrared signal from 
the infrared transmitter 60. The infrared transmitter 
20 60 may comprise an infrared diode or other appropriate 
infrared transmitting device. Such infrared signals 
are sensed by an infrared (IR) sensor 72 forming a part 
of the audio receiver 50, as shown in Fig. 3. A 
detailed example of the audio receiver 50 is shown in 
25 Fig. 5. 

Referring to Fig. 5, the infrared sensor 72, which 
may comprise any appropriate form of infrared sensor 
such as those typically used in remote control devices, 
provides the received signal for amplification by an 

30 amplifier 74 and filtering by a filter 76 . The output 
of the filter 76 is applied to a phase-locked loop 
receiver 78 which demodulates the frequency modulated 
signal by converting it to a voice signal. The phase- 
locked loop receiver 78 is of conventional phase -locked 

35 loop (PLL) configuration, and includes a phase detector 
80 coupled to the filter 76 and having an output and a 
second input coupled in a loop which includes a loop 
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filter 82 and a voltage controlled oscillator (VCO) 84. 
The output of the- phase-locked loop receiver 78 is 
filtered by an audio filter 86 and amplified by an 
audio amplifier 88 to provide the demodulated voice 
signal. The phase-locked loop receiver 78 also 
provides a lock detect signal, which indicates when a 
transmitted signal is present in the audio receiver 50. 

The phase- locked loop receiver 78 is described 
herein by way of example only, and it should be 
understood that other demodulation receivers can be 
used as desired. For example, a super-heterodyne 
receiver can be used in applications posing more 
stringent requirements. 

Voice control of the electronic device 14 within 
the voice and pointer operated system 10 is made 
possible by first voice training the collection of 
reference digital voice templates in the voice 
recognition circuit 26 in accordance with the user's 
voice. Such templates are collected in the same manner 
as described in co-pending application Serial No. 
07/915,112. The LCD display 44 of the device 14 is 
used to provide displays which prompt the user by 
requesting the needed words. The user responds, with 
the push-to-talk switch 21 being pushed, by speaking 
the prompted word into the microphone 20 of the pen 12 . 

As previously noted, the electronic device 14 in 
the example of Pig. 1 comprises a personal digital 
assistant (PDA) capable of input and retrieval of a 
phone directory. The LCD display 44 of the device 14 
is employed to display a list of names and phone 
numbers. By touching a button labeled "add", displayed 
on the LCD display 44 with the tip region 18 of the pen 
12, a blank entry with boxes for the name, voice 
training, country code, area code, phone numbers, 
address and notes is placed at the end of the list. 
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. The user is also given the option of adding additional 
phone numbers and alternate addresses per entry, if 
desired. The user may randomly select fields to input 
data, by touching such fields with the pen 12. For 
5 example, if the user touches a "name 11 box on the LCD 

display 44 with the pen 12, the name is input using tbe 
keyboard input 38, or by voice, spelling each word. 
The user can then touch a "voice" button and speak tbe 
person's name into the microphone 20 of the pen 12 in 

10 order to create a recording of the name. The user can 
then touch an "area code" box and speak the digits for 
the area code, or the user can touch the "phone number" 
box and speak the digits of the phone number, to 
complete the entry. The various fields are optional 

15 and can be selected in any order. If the user decides 
not to input data into a field, the field is left 
blank. A detailed example of this operation is 
provided hereafter in connection with Figs. 6 and 7. 
To retrieve a phone number, the user touches a 

20 "search" button on the LCD display 44 using the pen 12, 
and speaks the name of the person the user wishes to 
call. The person's name, phone number, and any other 
available information is immediately displayed on the 
LCD display 44, and the recording of the name is spoken 

25 back to the user for confirmation, using the audio 

circuit 40 and the speaker 42 shown in the arrangements 
of Figs. 2 and 3. Alternatively, the device 14 can be 
programmed to speak the phone number and other 
information or to dial the number if connected to a 

30 telephone. 

The flow chart of Fig. 6 illustrates the manner in 
which tbe user inputs all of the information normally 
found in an address book. For simplicity of 
illustration, the flow chart of Fig. 6 shows only one 

35 path through the various fields. However, it should be 
understood that the. user can input such information in 
any order desired. 
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Referring to the flow chart example of Fig. 6, in a 
first step 90 the user requests a new entry in the 
directory, and does so by touching a button labeled 
"add" with the pen 12 . The user then chooses to input 
5 the name first, by touching the "name" field with the 

pen 12, in a step 92. The "name" field is highlighted, 
and the user then types the person's name using the 
keyboard input 38, in a step 94. This highlights the 
"record" field. The user then holds the push- to- talk 

10 switch 21 on the pen 12 and speaks the person's name, 
in a step 96. 

Upon speaking the person's name, in the step 96, 
the "address" field is highlighted upon release of tbe 
push-to-talk switch 21. In a step 98, the user types 

15 in the address using the keyboard input 38. 

Alternatively, the user may record the address by 
speaking the address into the microphone 20 of the pen 
12 in a step 100. As a further alternative, the user 
can speak the digits of the street number into the pen 

20 12, in a step 102, and then type the street name using 
the keyboard input 38, in a following step 104. 

Following entry of the address, the "phone number" 
field is highlighted. The user may type the phone 
number using the keyboard input 38, in a step 106. 

25 Alternatively, the user may speak the digits of the 

phone number into the pen 12, in a step 108. As shown 
by a subsequent step 110, the address entry is 
automatically saved when an "enter" key on the keyboard 
entry 38 is pressed, or the push-to-talk switch 21 on 

3 0 the pen 12 is released, or one of the on-screen icons 
such as "save" or "new entry" is touched. 

The flow chart of Pig. 7 provides a more detailed 
example of the manner in which the user can randomly 
select fields and input information in any order. For 

35 simplicity of illustration, the flow chart of Fig. 7 is 
confined to entry of a phone number. Nevertheless, the 
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flow chart of Fig. 7 demonstrates the ease and 
flexibility with which the user may enter information. 

As illustrated by an elongated box 120 at the top 
of Fig. 7, the user has chosen to start a new phone 
5 number entry. The user may touch a "country" field, as 
illustrated in a step 122. If so, and as illustrated 
in a following step 124, the country code field is 
highlighted and the device 14 waits for the user to 
press the push- to- talk switch 21 on the pen 12 (again 

10 assuming the wireless example of Fig. 3) . By pushing 
the push-to-talk switch 21 on the pen 12 , in a step 
126, the device 14 is ready to interpret each spoken 
word and to append the result to the number in the 
country code field, until the push-to-talk switch 21 is 

15 released or the field is full, as illustrated in a step 
128. Release of the push-to-talk switch 21 results in 
the device 14 highlighting the country code field (step 
124) . If the field is full, the box 120 at the top of 
Fig. 7 is returned to. 

20 From the starting box 120, the user may also touch 

an "area code" field, in a step 130, and this 
highlights the area code field and waits for the user 
to press the push-to-talk switch 21 in a step 132. If 
the user presses the push- to- talk switch 21 in a step . 

25 134, the device 14 interprets each spoken word and 
appends the result to the number in the area code 
field, until either the user releases the button or the 
field is full, as shown in a step 136. Release of the 
push- to- talk switch 21 results in the highlighting of 

30 the area code field (step 136) . If the field is 
filled, the starting box 120 is returned to. 

From the starting box 120 at the top of Fig. 7, the 
user may also touch a "phone number" field in a step 
138, and this highlights the phone number field and 

35 causes the device 14 to wait for the user to press the 
push-to-talk switch 21, in a step 140. If the user 
then presses the push-to-talk switch 21, in a step 142, 
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the device 14 interprets each spoken word and appends 
the result to the number in the phone number field, 
until the user releases the push-to-talk switch 21 or 
the field is full, as illustrated in a step 144. If 
5 the user releases the push- to- talk switch 21, the phone 
number field is highlighted (step 140) . If the field 
is filled, the starting box 120 is returned to. 

As shown by the various lines and arrows in Fig. 7, 
from the step 124 in which the country code field is 

10 highlighted, the user may move to the step 130 or the 

step 138 for highlighting of the area code field or the 
phone number field respectively. Similarly, from the 
step 132 in which the area code field is highlighted, 
the user can move to either the step 13 8, or to the 

15 step 122 in which the country code is highlighted. The 
step 140 in which the phone number field is highlighted 
can also result in the user moving to either the step 
122 or the step 130. This illustrates the flexibility 
of the system and the fact that fields can be selected 

20 and information input thereto randomly and in any 
order . 

It is possible for two or more users to use the 
same electronic device 14 . Each such user trains his 
or her voice, with the resulting templates being stored 

25 separately from the templates of the other user or • 
users. Thereafter, and in accordance with one 
technique, a particular user initiates use of the 
device 14 by speaking his or her name, at an 
appropriate word group, and this results in the device 

30 14 using that person's voice templates. Once the name 
is recognized, the LCD display 44 shows the user's name 
and uses that person's voice templates, until the name 
is changed. Even if the device 14 is turned off, the 
device continues to identify that person and his or her 

35 voice templates when again turned on. This technique 
for use of the electronic device 14 by multiple users 
has the advantage that the electronic device 14 need 
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not have components added thereto to support the 
multiple voice capability. Also, the users do not have 
to remember a number on a switch or other device 
corresponding to their voice. However, the word groups 
must be increased by addition of more words, and it is 
difficult to constantly determine which user's voice 
templates are currently being used. 

Accordingly, an alternative approach to 
accommodating multiple, users of the electronic device 
14 involves the addition of a multiple position switch 
to the device 14 . Positioning such switch to a number 
corresponding to a particular one of multiple users 
results in the device 14 using that person's voice 
templates . 

While various forms and modifications have been 
suggested, it will be appreciated that the invention is 
not limited thereto but encompasses all expedients and 
variations falling within the scope of the appended 
claims . 
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WHAT IS CLAIMED IS: 

1. A voice and pointer operated system conprising 
the combination of: 

a pen responsive to a voice command for providing a 
5 voice signal; and 

a device having a pointer- responsive area thereon 
and including voice recognition means for recognizing 
the voice signal and producing an action routine 
denoted thereby. 
10 2. A voice and pointer operated system in 

accordance with claim 1, wherein the pen has a tip 
region for interacting with the pointer- responsive area 
of the device and a microphone responsive to the voice 
command . 

15 3 . A voice and pointer operated system in 

accordance with claim 1, wherein the pointer- responsive 
area of the device comprises a touch screen. 

4 . A voice and pointer operated system in 
accordance with claim 1, wherein the pen is 

20 electrically coupled to the device to provide the voice 
signal thereto . 

5 . A voice and pointer operated system in 
accordance with claim 1, wherein the pen includes means 
responsive to the voice command for transmitting a 

25 representation of the voice command, and the device 
includes means responsive to the transmitted 
representation of the voice command for providing the 
voice signal. 

6. A voice and pointer operated system in 
3 0 accordance with claim 5, wherein the means for 

transmitting includes an infrared transmitter and the 
means responsive to the transmitted representation 
includes an infrared sensor. 
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7. A voice and pointer operated system in 
accordance with claim 5, wherein the pen includes means 
for converting the voice command into an analog signal 
and means for modulating the analog signal to produce a 

5 transmission signal, and the device includes means for 
demodulating the transmission signal . 

8 . A voice and pointer operated system in 
accordance with claim l f wherein the voice recognition 
means includes a reference memory for storing a 

10 plurality of reference voice templates, a program 

memory for storing a control program, and a processor 
coupled to the reference memory and the program memory 
for generating an incoming voice template in response 
to each voice signal produced by the means for 

15 producing a voice signal corresponding to the 

transmitted representation, and for executing the 
control program to determine whether the incoming voice 
template is substantially equivalent to one of the 
reference voice templates, and for selecting one of a 

20 plurality of action routines based on the incoming 
voice template . 

9. A method of controlling, by voice command and. 
by pointing, a control device, comprising the steps of: 

pointing to a specific area of the control device 
25 to interact with and input data into the control 
device; and 

responding to a voice command by providing a 
corresponding voice signal to the control device. 

10. A method of controlling in accordance with 

30 claim 9, wherein the steps of pointing and responding 
are carried out simultaneously. 

11. A method of controlling in accordance with 
claim 9, comprising the further steps of responding to 
the voice signal at the control device by performing 

35 voice recognition of the voice signal and providing an 
action routine to the control device in accordance with 
the voice recognition performed on the voice signal. 



WO 95/25326 PCT/US95/02921 



- 22 - 

12. A method of controlling in accordance with 
claim 9, wherein the step of responding to a voice 
command by providing a corresponding voice signal to 
the control device comprises the steps of converting 

5 the voice command, transmitting the converted voice 
command to the control device and converting the 
transmitted converted voice command into a voice signal 
at the control device . 

13 . A pen for interacting with a touch screen, 
10 comprising the combination of: 

an elongated pen having a tip region for 
interacting with a touch screen; and 

means contained within the pen and responsive to a 
voice command for providing a voice signal. 
15 14 . A pen in accordance with claim 13 , wherein the 

means for providing a voice signal comprises a 
microphone . 

15. A pen in accordance with claim 14, wherein the 
microphone is mounted at an end of the elongated pen 

20 opposite the tip region. 

16. A pen in accordance with claim 13, wherein the 
means for providing a voice signal includes means 
responsive to the voice command for transmitting a 
corresponding signal . 

25 17. A pen in accordance with claim 13, wherein the 

means for providing a voice signal includes means 
responsive to the voice command for producing a 
corresponding analog voice signal, means for modulating 
the analog voice signal and means for transmitting the 

30 modulated analog voice signal. 
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18. A device for operating in response to a voice 
signal and to interaction with a pen comprising the 
combination of : 

means responsive to the voice signal for performing 
5 voice recognition thereon; 

means responsive to the voice recognition for 
producing an. act ion routine within the device; and 

means responsive to interaction with a pen for 
performing a function with the device. 
10 19. A device in accordance with claim 18, wherein 

the means responsive to interaction with a pen includes 
a touch screen mounted on the device . 

20. A device in accordance with claim 18, further 
comprising means for sensing a transmitted modulated 

15 audio signal and means for demodulating a sensed signal 
to produce the voice signal. 

21. A method of making entries in a directory- 
stored in an electronic device, the electronic device 
including a display having a plurality of touch- 

20 responsive fields and means for recognizing words 
spoken by a user, comprising the steps of: 
touching a name field on the display; 
manually entering a name into the device; 
speaking the name to enter it into the device; 
25 entering an address for the name into the device; 

and 

entering a phone number for the name into the 
device . 

22. A method in accordance with claim 21, wherein 
30 the step of touching a name field is carried out using 

a pen, and the step of speaking the name comprises 
speaking the name* into the pen so that the pen 
transmits a corresponding voice signal to the device. 

23. A method in accordance with claim 22, wherein 
35 the step of entering an address for the name comprises 

speaking the address into the pen so that the pen 
transmits a corresponding voice signal to the device. 
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24. A method in accordance with claim 22, wherein 
the step of entering a phone number for the name 
comprises speaking the phone number into the pen so 
that the pen transmits a corresponding voice signal to 
5 the device . 
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